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Claims 

1. A method for operating a system-resource-based multi-modal input fusion, the 

method comprising the steps of: 
5 receiving a plurality of user inputs; 

determining an amount of system resources available; and 

creating sets of similar user inputs, wherein a number of similar user 

inputs within a set is based on the amount of system resources available. 

10 2. The method of claim 1 further comprising the steps of: 

converting the plurality of user inputs into Typed Feature Structures 
(TFSs); and 

wherein the step of creating sets of similar user inputs comprises the step 
of creating sets of similar TFSs, wherein the number of TFSs within a set is based 
15 on the amount of system resources available. 

3. The method of claim 2 wherein the step of converting the plurality of user 
inputs into Typed Feature Structures comprises the step of converting the plurality 
of user inputs into a plurality of attribute value pairs and confidence scores. 

20 

4. The method of claim 2 wherein the step of creating sets of similar TFSs 
comprises the step of creating sets of similar TFSs, wherein a TFS is included in a 
set if it has a content score greater than a threshold, wherein 

25 ContentScore(TFS)= f( N, N A , N R , N M , CS(i)| i= i N ), 

where 

N= number of attributes in TFS, 
30 N A = number of attributes in TFS having a value, 

Nr = number of attributes in TFS with redundant values, 

Nm = number of attributes in TFS with missing explicit reference, and 

CS(i) = confidence score of the I th attribute of TFS. 
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5. The method of claim 2 wherein the step of creating sets of similar TFSs 
comprises the step of creating sets of similar TFSs, wherein a TFS is included in a 
set if it has a context score greater than a threshold. 

5 6. The method of claim 5 wherein the step of creating sets of similar TFSs 
comprises the step of creating sets of similar TFSs, wherein a TFS is included in a 
set if it has a context score greater than a threshold wherein 

ContextScore(TFS)=h(D m , RS(TFS,TFS m )) 

10 

where 

D m = number of turns elapsed since receiving TFS m from a modality 
RS = Relationship Score between TFS (current input) and TFS m 
1 5 TFS m — a TFS received D m turns ago. 

7. The method of claim 1 wherein a number of sets created is based on the amount 
of system resources available. 

20 8. The method of claim 1 wherein the step of receiving the plurality of user inputs 
comprises the step of receiving a plurality of multi-modal user inputs. 

9. The method of claim 1 wherein the step of determining the amount of system 
resources available comprises the step of determining an amount of memory or 

25 processing power available. 

10. The method of claim 1 wherein the step of creating sets of similar user inputs 
comprises the step of creating sets of similar user inputs, wherein a user input is 
included in a set if it has a content score greater than a threshold. 

30 

1 1 . A method for operating a system-resource-based multi-modal input fusion, the 
method comprising the steps of: 

receiving a plurality of user inputs; 

determining an amount of system resources available; and 
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creating sets of similar user inputs, wherein a number of similar user 
inputs within a set is based on the amount of system resources available, and 
wherein a number of sets created is limited based on the amount of system 
resources available. 

5 

12. The method of claim 1 1 further comprising the steps of: 

converting the plurality of user inputs into Typed Feature Structures 
(TFSs); and 

wherein the step of creating sets of similar user inputs comprises the step 
10 of creating sets of similar TFSs, wherein the number of TFSs within a set is based 
on the amount of system resources available. 

13. The method of claim 12 wherein the step of converting the plurality of user 
inputs into Typed Feature Structures comprises the step of converting the plurality 

15 of user inputs into a plurality of attribute value pairs and confidence scores. 

14. The method of claim 11 wherein the step of receiving the plurality of user 
inputs comprises the step of receiving a plurality of multi-modal user inputs. 

20 15. The method of claim 1 1 wherein the step of determining the amount of system 
resources available comprises the step of determining an amount of memory or 
processing power available. 

16. An apparatus comprising: 
25 a plurality of modality recognizers receiving a plurality of user inputs; and 

a semantic classifier determining an amount of system resources available 
and creating sets of similar user inputs, wherein a number of user inputs within a 
set is based on the amount of system resources available. 

30 17. The apparatus of claim 16 further comprising: 

segmentation circuitry converting the plurality of user inputs into a 
plurality of Typed Feature Structures (TFSs); and 

wherein the semantic classifier creates sets of similar TFSs, wherein the 
number of TFSs within a set is based on the amount of system resources available. 
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18. The apparatus of claim 17 wherein the number of sets created is limited based 
on the amount of system resources available. 

19. The apparatus of claim 16 wherein the number of sets created is limited based 
5 on the amount of system resources available. 



